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ABSTRACT 

The authors describe in this paper the progress achieved to-date with the reengineering of the Earth Observing System 
(EOS) Data and Operations System (EDOS), the experience gained in the process and the ensuing reduction of ground 
systems operations costs. The reengineering effort included a major methodology change, applying to an existing 
schedule driven system, a data-driven system approach. 


1. INTRODUCTION 

The EDOS system currently supports the following five missions: Terra, Aqua, Aura, ICESat and EO-1. Support for the 
future OCO mission is planned for a start in December 2008. EDOS is a distributed system. Its functions are to capture 
the X-Band science downlinks data at four ground stations, to transfer the data to a central location referred to as the 
Level Zero Processing Facility (LZPF), to generate, archive and distribute level zero products to end-users ? 
organizations around the world. 
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Figure 1. EDOS Ground System Functions, Data Flow and Data Volume 






Evolving the system to reduce budgets, increase efficiency and service new missions has been a challenging task. In 
June 2005, at the 6th RCSGSO symposium in Darmstadt, the authors described several initiatives to reduce the 
workload on operations personnel. It was projected then that a data-driven system would bring about savings in the 
operations of the system and at the same time increase system reliability. 

The data-driven initiative was then pursued actively and the solution was deployed in June 2006 under Phase 1 . The 
current system no longer relies on schedule to capture data and the data processing has been completely automated end- 
to-end. Under Phase 2 currently under development and targeted for deployment by the end of the fall 2007, all the 
hosts/servers will be replaced as well as the special hardware providing for added flexibility, performance and several 
new functions, services. Finally in the proposed plan for Phase- 3 for the 2008-2009 time frame the authors will address 
system changes associated with products deliveries and products quality assurance automated verification. 


2. GROUND SYSTEM OVERVIEW 

The diagram in Figure 1 depicts the ground stations and the corresponding antennas with one EDOS Data Driven 
Capture Device (DDCD) connected to service the incoming data stream. The DDCDs supplied by Kongsberg Spacetec 
are part of the EDOS Ground Station Interface Facility (GSIF). EDOS receives on average seventy-one contact 
sessions for a total of 445 Gigabytes of data per day. The incoming data stream triggers the DDCD to capture process 
and forward the data to the LZPF. The data is also stored into a local archive at the site for 30 days. 

At the LZPF, the received playback data from the stations via the IP networks is ingested by the High Rate Service 
Processors (HRSP) for service processing, generating files ready for distribution to users and/or to the Product Data 
Processors (PDP). The PDPs then generate the Product Data Sets (PDS) and/or Expedited Data Sets (EDS) for 
distribution. The playback data files are stored into an archive for some 20 days, should reprocessing of some data be 
required. 

On average, the HRSPs deliver about 1,188 Rate Buffered Data (RBD) files daily for a total of 203 Gigabytes and the 
PDPs deliver 895 PDSs/ EDSs for a total of 281 Gigabytes. EDOS initiates the transfer of the Rate Buffered Data 
(RBD) within 5 minutes after LZPF receipt of all data for the contact session. EDOS delivers the PDSes within 24 
hours of receipt of all appropriate input data. Similarly the EDSes are delivered within 60 minutes to the users. 

EDOS also initiates the transfer of Customer Operations Data Accounting (CODA) status data for data quality and 
statistics monitoring to the control centers within 1 second of receiving data and continues transfers in 5 seconds 
intervals. 

3. EDOS DATA-DRIVEN - PHASE 1 JUNE 2006 

3.1 Design 

The data-driven solution was presented last June at the SpaceOps conference in Rome (2006) in a paper titled 
“Autonomous multi-missions, multi-sites X-band data capture systems”, at a time when the solution was actually being 
deployed on the systems. Kongsberg Spacetec’s presentation at the 7 th RCSGSO symposium will address the 
engineering design of the DDCD device, the solution and future enhancements. 

EDOS custom application software and processes running at the LZPF on Silicon Graphics and IBM AIX servers had 
to be modified to remove all the schedule dependencies leading to a major system simplification. 

3.2 Features and Benefits 

To meet the goal of increasing efficiency and reducing operation costs while meeting the original performance 
requirements, the system had to perform the following: 

• Autonomous capture of high rate data for supported missions at each antenna with no schedule input using 
new Kongsberg-supplied Data-Driven Capture Devices (DDCD) 

• Automatic transfer of high rate data to LZPF with no schedule input 

• Automatic “reconciliation” of data transferred to LZPF 

® Automatic LZPF data product generation and distribution 



• Automation of EO- 1 data capture and distribution 

® Use of new LZPF Network Attached Storage to retain data received for 25 days 

• Re-processing requests serviced from the LZPF Network Attached Storage 
® Use of Nagios for real-time monitoring of all DDCDs 

The expected resulting benefits would then translate into 

• Reduction of EDOS operation costs by automating routine data capture and processing 

• Reduction of missed passes and reduction of FOT rescheduling efforts 

® Streamlining of the EDOS system architecture by reducing software and database complexity 
® Positioning the project for Phase 2 EBOX technology upgrade, associated automation and re-engineering 
efforts 

• Automation of the manual EO- 1 processing 

3.3 System Transition 

The system transition took place on June 14 th , 2006 and within 24 hours; all sites were converted to the data-driven 
solution with minimum impact to the latency on product deliveries for that day. Operations personnel directed the 
transition with the support of engineering personnel at the ground sites. The strategy involved migrating 
progressively the functions to the data-driven system as Operations personnel continued the processing of the 
mission data on the backup system through the day. 

3.4 Problems experienced and Resolution 

Prior to deployment, the data-driven solution was evaluated through systems tests and acceptance testing phases on 
a separate test system string end-to-end. Most known bugs were resolved by the KSPT and EDOS developers and 
the system was accepted for deployment into operations. 

Several intermittent problems were experienced shortly after transition. 

Captured files on the DDCD had incorrect AOS and LOS times, creating a situation where the data would not be 
included in products creation at the LZPF and thus contribute possibly to incomplete products. Data overflows 
began to appear after several days and it was realized that there was a memory leak associated with the middleware 
task. Files transfer failed the reconciliation using the checksum. At times, the processing queue would be affected 
when an item would be in a hung status. Finally, there were other problems with the EDOS backend application 
software, service processing, product data processing and communication tasks which were fixed as they were 
uncovered. 

All the problems above have since been resolved by the respective developer parties. 

3.5 Performance analysis to-date 

The replay factor was the parameter of choice to measure the impact of the data-driven solution on EDOS 
operations. 

The replay characterizes the manual process required by operations to re-ingest the data into the DDCD from the 
ground station Line Outage Recorder (LOR). The analysis of the replays statistics to assess the design 
improvement or lack of, when operating the system in a data-driven mode, is based on the data collected at the 
WSC ground station, where EDOS captures the Terra X-band science downlinks at 150 Mbps from TDRS every 
45 minutes or so. Alaska, Norway and Wallops ground stations have shown more variability because of natural 
events including winds, flood and /or power loss, com equipment malfunction, 

The data shown in Figure 2 represents the average monthly replays performed by EDOS operations since July 
2004. The data is shown for three time periods, the first extending from July ’04 through May 2006, the second 
from June 2006 through December 2006 and the last from January 2007 through April 2007. 




Figure 2. Terra Monthly Average Replay Statistics 


Until June 2006, the system was in a schedule driven mode with an average of 22 replays per month, mostly due to 
problems with system scheduling of resources. The second period shows an average of 18 for the data-driven 
system, not a significant improvement due to problems previously mentioned in section 3.4. Then, finally after all 
the fixes were implemented, the average dropped dramatically to 1.5. 

It took about 6 months to reach a new all time high in system dependability. At no time since June 2006, has the 
data-driven system failed to ingest the incoming data stream and to identify correctly the mission data at all four 
ground sites. A performance level has now been achieved to enable further cost reduction in operations support. 


3.6 Lessons Learned 

The data-driven solution has brought about a major system simplification. Since it was deployed, there have been 
at least 26,000 spacecraft contact sessions and all missions contacts were correctly identified by the data capture- 
ingest task. This is the solution of choice given the constraints imposed on EDOS in servicing the missions. 

A few unexpected glitches on the software code created initially modifications to the operators’ procedures where 
operations personnel had to pay careful attention to make sure the problems did not contribute to data losses. A 
lesson learned here is to never expect the software to be error free. The software patches deployed afterwards have 
made this system highly dependable as suggested by the replay number statistics since January 2007 for WSC, out 
of 3,600 passes, there were less than 5 replays. 

There are risks associated with major system reengineering changes such as converting a schedule driven system to 
a data-driven system. Best mitigation approach still is a separate test system with ample testing time to uncover the 


defects. Where possible, the best approach suggests also a period of parallel operations before the system is placed 
on line. The EDOS systems have now reached a new high level of dependability, flexibility and autonomy. 

4. EDOS DATA-DRIVEN - PHASE 2 November 2007 

EDOS’ evolution continues based on the successful Phase 1 experience to enhance further the data-driven solution 
and to bring into reality the concept of the EDOS-in-a-Box (EBox) where, with a single piece of hardware, EDOS 
can capture, process, and deliver science data products. Refer to Fig 3. 

• The EBox supplied by KSPT (HP ML570) configured in the “send” mode (EBox-S) deployed at the 
GSIFs replaces the DDCD and can be configured to perform data driven capture, CADU transfer, service 
processing, production data processing and delivery. 

• The EBox configured in the “receive “mode (EBox-R) deployed at the LZPF replaces both the HRSP and 
PDP processors. 

• The Monitor and Control System (M&CS) will provide via enhanced GUIs a centralized and streamlined 
view of health & status of EDOS components, processing, enhanced anomaly reporting, and elimination 
of tracking forms 

The EDOS phase 2 architecture extends the performance capabilities of the existing C5.0 system and eliminates all 
operator interaction during nominal support and integrates better with Flight operations support. 



Figure 3. EDOS Phase 2 Architectural Overview 




4.1 Phase 2 Key Design Guidelines 


The following key design points are guiding the current effort: 

- Standardization of all EDOS operator screens to a unified set based on JAVA and accessible through 
a web browser. An additional external web server will allow the ground stations, FOT, and science 
teams to non-intrusively observe the EDOS capture and production status at any time and take 
necessary corrective actions if needed 

- Operation of the system requiring intervention only on system anomalies and not during nominal 
cases via a central control and monitoring system 

- Incorporation of new FEP design that increases EDOS spacecraft downlink capture rate up to 500 
Mbps and allows in-system reconfiguration/ upgrades via FPGA image files 

Allowing to create level zero products right at the ground station and option to deliver data in real- 
time; “EDOS-in-a-Box” capability to support Remote Science Processing for disaster recovery 
Consolidation of the High Rate Service processors and the Production Data Processors into single 
units, reducing the amount of hardware, replacement of obsolete SGI technology and use of a single 
operating system (LINUX) to simplify system administration. Reduce the number of science 
processing machines in the LZPF from 7 per configuration to 3 

- Implementation of a GMSEC bridge to provide status data feed to external automation engines. 

- Upgrades hardware at ground stations to the latest technology reducing hardware obsolescence risks. 

- Automated Session and Product Status reporting. 

- Capability to prioritize traffic between the GSIFs and the LZPF high rate lines via QoS 
implementation 

- Archive data storage based on disks and no longer on tapes 

4.2 Phase 2 Development Status 

The integration of the components into a test system is currently ongoing at GSFC. A test string has been 
configured to incorporate the hardware platforms, the application software and other software components as they 
become available from the respective developers. 

The KSPT team efforts are focused in delivering the EBox-S, the EBox-R and the M&CS systems including the 
MEOS software and the middleware components. To mitigate a delay incurred with the production of the new 
ingest 10 2100 boards, a pair of prototype boards will be shipped earlier and inserted in the test system to enable 
the evaluation of the data ingest process at the earliest opportunity. 

The EDOS developers concentrate their effort on the configuration and integration of the EBox-R which performs 
the service processing, the production data processing and the product distribution. The legacy software has been 
ported to run on a new platform and o/s environment under Linux Suse 10.1; the porting of the software from 
Silicon Graphics servers to newer HP platforms was a major conversion task. 

Additionally both teams are finalizing the GUI screens and functionality to support streamlined operations. It is 
anticipated that the integration testing and acceptance testing would not complete before October 2007. Pending 
on the confidence level obtained during the testing and the amount of parallel operations required, the system 
would not be deployed before mid-November 2007 at the earliest. 

4.3 Post Phase 2 Items 

The EDOS system will be modified to support changes evolving from Auto-Ops mode of operation of the 
spacecrafts. Because EDOS performs the service processing per individual contact sessions, a technique to recover 
the CCSDS packet contained in multiple frames across contact sessions will be developed. 

For the OCO mission, there is a need to develop a capability to extract in real-time several virtual channel data 
from the X-band downlink as the data is ingested on the ground and to forward the data with the least amount of 



delay to an end-destination. The capability to stream data across the networks for several virtual channels is also 
new and will be provided shortly after the C5.1 delivery. 


5. EDOS DATA-DRIVEN - PHASE 3 November 2008 

Under the current operational configuration and with the C5.1 configuration as well, EDOS delivers products 
autonomously by using a “push” method depositing the data into directories on the end-users’ servers. A preferred 
way to distribute the products would be for EDOS to push all products to a central server and let the users pull their 
respective data from the server. The data would remain available on the server for 10 days. This would reduce the 
workload on the EBox-R servers which receive the data, process it and generate the data products. 

At times, due to problems at the ground station, communications lines problems, the data may not be transferred 
back to the LZPF in the correct time order. That situation then may trigger prematurely builds of products and, as a 
result, generate incomplete products. With the GSFC Mission Services Evolution Center (GMSEC) automated 
tools, EDOS plans to develop automated “quality assurance” processes to detect and report on these anomalies. 


6. SUMMARY 

The data-driven solution deployed under Phase 1 simplifies operations in a major way and the methodology has 
been validated. Credits go to the KSPT and the EDOS development teams to resolve the bugs on a timely basis. 

Under Phase-2, all computer/ hosts processors will be replaced with a single type of platforms, one o/s and 
operations personnel focus on anomalies reporting only. This will be a major system transition as almost every 
single piece of hardware in the system will be replaced. 

Under Phase 3, the delivery of the products by EDOS to a central server will be more in line with an optimized 
approach for products distribution as end-users systems would pull the products from the central server. 



Figure 4. EDOS Staffing Trend 


Figure 4 depicts the downward trend with the staffing of personnel supporting the workload of 5 missions and in the 
planning of adding the OCO mission and support for ALOS. As more missions make use of the existing antennas 
supported by EDOS at the ground stations and the high rate networks in place to deliver the data, significant savings 
continued to be achieved on the individual programs. 

Figure 5 depicts the EDOS total cost profile referenced to Fiscal Year 2002 



Figure 5. EDOS Total Cost per Fiscal Year 
The following comments interpret the curve trends in Figure 5 

• In 2003, the drop reflects a decline in system development costs. 

• In 2004, operational efficiencies are realized leading to a reduction of operations, management and 
administrative positions. 

• In 2005, the reduction in mission engineering and project troubleshooting positions continued as on-orbit 
missions stabilized into routine operations and there was limited new mission development work. 

• In 2006, the NAS implementation eliminated maintenance costs of Ampex tape drives and tape handler 
positions. Further operational efficiencies are achieved as the system stabilized and became more automated 

The consolidation of functions into a reduced number of computers and the use of a single operating system contribute 
to the reduction of maintenance costs. The added high system reliability reduces as well the need for technical 
assistance at the remote sites. As an example, the data-logger at WSC has been operating in a LOR mode with no site 
personnel support needed since October 2005. 
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